A lexical analyzer may analyze input computer programs by assigning a token to each of the lexemes in the ordered string of text that constitutes the computer program. A lexeme is a distinct group of consecutive characters (e.g., “func”, “=”, “325”, “;”, etc.), while a token is an identification of a type, category, or classification of a lexeme (e.g., “variable”, “integer”, “assignment”, “comma”, etc.).
Such a lexical analyzer may be included, together with a syntax analyzer, as part of a parser. The lexical analyzer may generate a stream of tokens and/or token-lexeme pairs, while a syntax analyzer may check the output of the lexical analyzer for syntax errors. A parser is often used as part of a compiler. A parser may also be used as a syntax checker, for example in a code editor or integrated development environment (IDE), by analyzing a program and notifying a programmer if there are any syntax errors.